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EVALUATION SUMMARY 



The Comprehensive Instructional Management system (CIMS) 
Mathematics project is a teacher- developed mathematics curriculum 
for kindergarten through grade seven. It is administered through 
the Office of Program and Curriculum Development (O.P.C.D.) of 
the New York City Board 6f Education, and includes teacher 
manuals for each g^ade, corresponding criterion-referenced tests, 
and a computerized test scoring and reporting system. Piloted in 
five New York City community school districts (CS.D.s) in 
1982-83, the project operated in 1988-89 in 26 school districts, 
and in 15 of them it served as the main districtwide mathematics 
curriculum. 

The original CIMS-Mathematics model included a 
district-based computer management system to score tests and 
generate management reports. In 1986-87, the project began a 
school-based version of the management system (PC-CIMS) in two 
districts. Because of the success of PC-CIMS, an improved 
version was initiated in three additional districts in 1988-89. 

The objectives of the CIMS-Mathematics project in 1988-89 
were to: 

• effect growth in student mathematics achievement; 

• continue to introduce and support the use of the CIMS- 
Mathematics curriculum in New York City districts and 
schools ; 

• expand the PC-CIMS model of project operation into 
additional districts and schools, and support the ongoing 
operation of existing centralized management systems; 

• initiate, enhance, and expand staff development programs for 
teachers and supervisors in support of CIMS, especially 
staff development for principals, supervisors, and teachers 
in the instructional and supervisory uses of the PC 
management systems; 

• promote district- and school-based management of project 
activities; and 

• revise curriculum manuals, student workbooks, and tests for 
grade seven. 

The 1988-89 evaluation focused on the operation and impact 
of the program in a sample of five districts, chosen to reflect 
varied lengths of participation in the project and different 
models of the computer management system: three of the five 
original pilot districts, of which two continued to use the 
original centralized systems, and two PC-CIMS districts, each 
using a different version of the PC system. A second part of the 
evaluation documented the operation and use of PC-CIMS in a 
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sample of schools in the two PC districts. It aimed to examine 
in greater detail the nature and effectiveness of attempts to 
change instructional and supervisory practice through the use of 
the computer management system , and to identify the arrangements 
that promote the greatest degree of self-sufficiency on the part 
of district and school staff in managing project activities. In 
this effort, the evaluation drew on a number of data sources: 
interviews with district, school, and project staff; observation 
of project activities and review of project records; and student 
mathematics achievement data for spring 1988 and for 1989. 

Among the major findings were the following: 

• In all districts examined, CIMS was associated with 
continued growth in mathematics achievement from 1988 to 
1989. For the five districts, there was an overall 
statistically significant mean gain of 21.0 scale score 
points ( s. D. =27. 5) f which represents an educationally 
meaningful gain. Students' mean gains ranged from 25.2 
sea?** score points ( S.D. =28.6) in C.S.D. 30, to 17.0 scale 
score points (S.D.=27.2) in C.S.D. 32. All district mean 
gains were statistically significant, and in C.S.D. s 1, 18, 
and 30, they were educationally meaningful. 

• All of the sampled PC schools reported mean gains ranging 
from 34.1 scale score points ( S.D. =22.6) to 26.3 scale score 
points ( S.D. =2/.6) . All of these gains were statistically 
significant and educationally meaningful; in all cases, mean 
gains in the sample schools outstripped those or the 
district as a whole. 

• An analysis of the correlation between percent of CIMS 
objectives mastered and performance on the citywide 
mathematics achievement test (MAT) for 1,422 pupils in 
grades three through six in C.S.D. 30 revealed correlations 
ranging from 0.79 to 0.84. These were very high levels of 
correlation and indicated that mastery of objectives as 
measured by the CIMS tests was a very good predictor of 
performance on the mathematics achievement test for each 
grade. Moreover, since the MAT is customized to New York 
City and New York State curricula mastery of CIMS 
objectives is associated with mastery of these curricula 

• Efforts to promote management of project activities by 
district staff were largely successful, as district 
mathematics coordinators began to take charge of the ongoing 
operation of project activities in their districts, assumed 
responsibility for continuing staff development in district 
schools, and were involved in selecting and preparing 
schools to join the project the following year. 

• CIMS liaisons in the schools, and to a lesser extent 
principals and other supervisors, monitored the use of the 
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curriculum and tests, provided support to teachers in 
implementing the program, encouraged spiral ing homework, and 
promoted coordination of remedial mathematics programs with 
CIMS. There was some variation in the extent to which 
supervisors other than the CIMS liaisons incorporated use of 
the management system into their supervisory activities. 

• Nearly all teachers in sampled schools reported that their 
supervisors had spoken to them about pacing the curriculum 
or tests, but only a third had discussed specific test 
results or suggested instructional strategies for teaching 
mathematics. On the whole, three-quarters of teachers found 
their school-based staff development very helpful (38 
percent) or moderately helpful (38 percent) , although there 
were large differences among the schools in teachers 1 
ratings of this assistance. Teachers were most satisfied 
with their school-based staff development where supervisors 
provided concrete forms of assistance, such as suggested 
strategies and practice items. 

• Sixty percent of teachers reported that they regularly 
reviewed the item analysis, the objective summary report, or 
both; only 16 percent of teachers reported that they rarely 
or never reviewed the test repcrts. Two-thirds of teachers 
said they regularly consulted a copy of the test when 
interpreting the test results. On the whole, teachers were 
more likely to reteach objectives as a result of test 
findings (41 percent) than to spiral homework assignments 
(17 percent) ; the remainder did both equally. 

• The highest percentages of teacher respondents reported 
using the test reports regularly for review or "Do Now" 
exercises in class (63 percent) , to assist 
paraprofessionals' work with students (54 percent), and to 
inform parents (54 percent) . Teachers were less likely to 
us* the reports for grouping students for instruction or 
honework (24 percent) , individualizing instruction or 
homework (21 percent), or peer tutoring (16 percent). 

• A large majority of teachers reported that they always (32 
percent) or often (47 percent) spiraled homework 
assignments, and nearly two-thirds of all teachers reported 
using the t^st reports for this purpose regularly. 

• There was a consistent finding of greater use of test 
reports, apart from grouping, and higher ratings of the 
usefulness of the management system in the C.S.D. 30 schools 
using the older PC-CIMS model than in the C.S.D. 17 schools 
using the newer PC system. This was true for both the 
first- and second-year C.S.D. 30 schools. 

• Commitment to continue the project was high on the part of 
the teachers interviewed: 45 percent were very committed 
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and 34 percent moderately committed to its continuation. 
Sixty-three percent of 30 respondents said that they were 
more committed to the program than the previous year. 

On the basis of the evaluation, the following 
recommendations are made: 

• Continue to expand the PC component of the project and staff 
development in support of it. 

• Focus efforts on training supervisors, as well as CIMS 
liaisons, to provide assistance with the management system 
and its use for instructional purposes. 

• Evaluate carefully the format and number of test reports and 
their usefulness for teachers and administrators; consider 
simplifying the newer PC reports. 

• Continue efforts to promote the use of the management 
reports for grouping students and individualizing 
instruction. 

• Help teachers to find alternate ways of teaching 
mathematical concepts through the use of the testing system. 

• Consider carefully teacher and supervisor comments in 
reviewing and revising the first grade curriculum. 

• Re/iew test formats to check for unclarity and ambiguity in 
the questions, especially in grade two. 

• Continue stressing use of the tests and trst reports for 
diagnostic purposes. 

• Assist teachers in finding ways to share the test reports 
effectively with parents, perhaps by linking them with other 
teachers in their schools who use them successfully for this 
purpose. 
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I. INTRODUCTION 

PROGRAM BACKGROUND 

The Comprehensive Instructional Management system 
(CIMS) -Mathematics project is a teacher-developod mathematics 
curriculum for kindergarten through grade seven, including 
criterion-referenced tests and a computerized test scoring and 
reporting system for managing mathematics instruction. The 
CIMS-Mathematics curriculum model was designed to incorporate all 
New York State and City mathematics instructional objectives and 
to be adaptable to individual districts' needs and resources. It 
was initiated in upstate New York in I97u, and prototype projects 
we^e introduced in several New York City community school 
districts (C.S.D.s) during the following decade. 

The current CIMS-Mathematics project was begun by the New 
York City Board of Education's Division of Curriculum of 
Instruction. The project is currently administered by the Office 
of Program and Curriculum Development (O.P.C.D.). It was 
developed in 1982-83 with a grant from the New York State 
Department of Education, and piloted the following year in five 
community school districts. Since then, use of the project has 
continued to expand throughout the New York City public schools. 
In 1988-89, CIMS-Mathematics curriculum manuals were in use in 26 
of the 32 New York City community school districts. Of these, 15 
had adopted the manuals districtwide in all schools and grades. 

The original CIMS-Mathematics model included the operation 
of a district-based computer management system to score tests and 
generate management reports. Because of the large commitment of 
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resources and personnel needed to operate such systems, the 
project introduced a management system in 1986-87 using 
school-based microcomputers, called PC-CIMS. PC-CIMS was begun 
in nine schools in two districts in 1986-87 and expanded to an 
additional seven schools the following year. As a result of the 
success of PC-CIMS, in 1988-89 the PC model was further expanded 
in the two original districts, and a hard-disk version allowing 
for greater efficiency and expanded functions was initiated in 
three additional districts. 

Staff development for teachers and supervisors in the use of 
the curriculum manuals, tests, and test reports has been an 
important part of the CIMS program in all phases of its 
implementation. Through the use of the computer management 
system, the project seeks not only to improve mathematics 
instruction, but to change and enhance the nature of mathematics 
supervision in the schools. When PC-CIMS was introduced, project 
staff assigned a CIMS staff developer to each of the 
participating districts on a permanent part-time basis. With the 
expansion of the FC model and the need to provide support to an 
increasing r.umber of districts, the project modified this 
arrangement to lessen the direct dependence of schools on CIMS 
staff and to broaden the role of district and school staff in the 
management of project activities. 
PROGRAM OBJECTIVES 

The major objectives of the CIMS-Mathematics program for 
1988-89 were the following: 
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• effect steady growth in student mathematics achievement; 

• continue to introduce and support the use of the CIMS- 
Mathematics curriculum manuals in New York City districts 
and schools; 

• expand the PC-CIMS model of project operation into 
additional districts and schools, and support the ongoing 
operation of existing centralized management systems; 

• initiate, enhrnce, and expand staff development programs for 
teachers and supervisors in support of CIMS, especially 
staff development for principals, supervisors, and teachers 
in the instructional and supervisory uses of the PC 
management systems; 

• promote district- and school-based management of project 
activities; and 

• revise curriculum manuals, student workbooks, and tests for 
grade seven. 

POPULATION SERVED 

During 1988-89, the CIMS-hathematics project operated in all 

schools from kindergarten through grade seven in 15 New York City 

community school districts. Of these, four were original pilot 

districts involved in the project since its inception six years 

ago (C.S.D.s 1, 18, 23, and 32), five had implemented the project 

for four years (C.S.D.s 5, 8, 15, 27, and 30), four had used the 

curriculum districtwide for three years (C.S.D.s 6, 10, 17, 21), 

and two were in their second year of districtwide implementation 

of the project (C.S.D.s 7 and 11). In addition, CIMS curriculum 

manuals were used in 1988-89 in selected schools and/or grades in 

11 other school districts. Depending on district decision, both 

general and special education students were served. 
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PROGRAM EVALUATION 

The 1988-89 evaluation of the CIMS-Mathematics project by 
cne Office of Research, Evaluation, and Assessment/Instructional 
Support Evaluation Unit (OREA/I.S.E.U. ) was in two parts. The 
first examined project activities in a sample of five districts, 
chosen to reflect varied lengths of participation in the project 
and the use of different computer management systems. Since the 
project had operated in these districts for a number of years, 
the evaluation focused on current issues in the ongoing 
implementation of project activities, and reported on the impact 
of the program on student achievement in mathematics for 1988-89. 
The second and main part of the evaluation sought to examine in 
greater detail the nature and effectiveness of attempts to change 
instructional and supervisory practice through the use of the 
computer management system, and to identify the arrangements that 
promote the greatest degree of self-sufficiency on the part of 
district and school staff in managing project activities. To 
this end, the evaluation documented the operation and use of 
Fc-CIMS in four schools in two districts, one implementing 
PC-CIMS for the. first time during the current year and one 
continuing the oparation of existing systems. Project staff 
development support to the schools, follow-up support by district 
and school personnel, and use of the management reports by 
teachers, principals, and assistant principals were examined. 
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SCOPE OF THE REPORT 

This report presents the results of the evaluation of the 
1988-89 CIMS-Mathematics project. Background information and 
goals for the year are reviewed in Chapter I. Major project 
components are described in Chapter II, and the evaluation focus 
and methodology are discussed in Chapter III. Chapter IV reports 
findings from parts one and two of the evaluation: part one 
presents findings on the operation of the project and its 
effectiveness in a number of districts implementing the program 
for varying lengths of time; and part two documents the operation 
of PC-CIMS in a sample of schools in two districts. Conclusions 
and recommendations are presented in Chapter V. Appendices A, B, 
and C show 1988-89 analyses of CIMS mathematics scale scores by 
district, CIMS mathematics N.C.E. scores by district, and CIMS 
mathematics scale scores and N.C.E. scores for sample schools, 
respectively. 
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II. PROJECT COMPONENTS 



STAFFING AND ADMINISTRATION 

Two co-directors were responsible for the overall 
administration of the CIMS project. Their duties included: 
introducing the project to districts and schools new to CIMS; 
assisting mathematics coordinators in developing and implementing 
staff development activities in the districts; coordinating the 
PC-CIMS project; overseeing revision of curriculum manuals, 
student workbooks, and tests; and documenting project 
activities. 

Four full-time CIMS staff developers, called "curriculum 
leaders, 11 conducted districtwide and in-school training for 
principals, assistant principals (A.P.s), and teachers on the 
basis of staff development plans made with the districts at the 
request of individual schools. In order to lessen the dependence 
of the districts and schools on one CIMS staff member, pairs of 
curriculum leaders dealing with computer- and curriculum- related 
issues were assigned to work with districts on a flexible basis. 
Requests for staff assistance from the schools were expected to 
be handled through the district mathematics coordinator. 
Curriculum leaders were also involved in the curriculum revision 
process. 

District mathematics coordinators functioned as the primary 
CIMS liaisons in the districts and were expected to take over 
management of project activities in their districts after an 
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initial year of training and support by CIMS staff. In 
preparation for this role, CIMS requested that mathematics 
coordinators of districts joining the project attend all staff 
development sessions conducted by CIMS in their district. After 
the first year, the mathematics coordinator was responsible for 
staff development for schools continuing in the project, while 
CIMS provided orientation and support for new schools. 
Additional responsibilities of the mathematics coordinators 
included ordering and distributing CIMS materials in the 
districts and schools, and providing feedback from teachers 
regarding curriculum revision. 

The central CIMS computer manager and Office of Data 
Processing Coordination (C.D.P.C.) staff were responsible for 
overseeing the operation of the district-based computer 
management systems and the in-school PC systems. Implementation 
of the ^C systems involved training school-based personnel in the 
operation of the computers, overseeing their installation and 
ongoing operation, and designing and updating CIMS software and 
management reports. District computer managers were responsible 
for the day-to-day operation of the district-based systems: 
updating student rosters; overseeing test delivery, scoring, and 
reporting; and maintaining hardware and software. In the PC-CIMS 
schools, computer aides or administrative assistants were 
responsible for maintaining student class data, scoring test 
answer cards, generating test reports, and coordinating the 
delivery of tests and reports. 

7 
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CURRICULUM 

CIMS-Mathematics curriculum materials include a two-part 
teachers manual for each grade and corresponding student 
workbooks. The teaching strategies suggested for each 
instructional objective make extensive use of manipulative 
materials and are intended to be supplemented by additional 
material and practice items from other resources. The CIMS 
material supports the cyclical organization of the New York Citv 
Scope and Sequence , which draws on research reportedly showing 
that students remember material more effectively when topics are 
reintroduced at various times rather than covered exhaustively 
once. To correlate with the cyclical nature of the Scope and 
Sequence, teachers are expected to "spiral" homework assignments. 
This means that each homework assignment should include review 
questions and preparation for future work, as well as cover 
material currently being taught. Ideally, a teacher should use 
the computerized test reports to prepare a spiraled homework 
assignment with four components: questions on the current lesson, 
questions on objectives missed during the past two weeks, 
questions on objectives missed earlier in the year, and review of 
a skill that will be needed for a future unit. 
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TESTS 

Integral to the CIMS curriculum are criterion-referenced 
tests, each covering three instructional modules and consisting 
of 25 to 45 multiple-choice items. There are 12 cumulative tests 
and four quarterly tests in grades two through six, and 27 
individual module tests in grade one. CIMS tests are not 
designed primarily for evaluation purposes, but are intended to 
be used along with the computerized test management system to 
diagnose student weaknesses and plan instruction and homework 
assignments. 

CO MPUTER MANAGEMENT SYSTEM 

Linked to the CIMS testing component in the original CIMS 
model was a district-based computer management system, whose 
function was to provide teachers with computerized test results 
for instructional planning, and to give administrators and 
supervisors information for monitoring mathematics instruction 
and making long-range educational decisions. This system 
continues to be used in two of the original pilot districts. The 
newer PC model uses in-school computers to score tests and 
generate reports. The original PC system in use in two districts 
generates a number of reports for each test administered: an 
item analysis of the entire test; a report of objectives mastered 
and not mastered by each student; and a summary report of 
objectives mastered by the class. The newer hard-disk system in 
use in three other districts has increased capabilities and for 
each test generates two item matrices, two instructional 
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objective matrices, and a class diagnostic report, which groups 
students according to objectives missed. An individual summary 
report, listing objectives missed by each student, accompanies 
the results of each quarterly test. 

Teachers are expected to review the test results with a copy 
of the test in hand to determine whether student errors are due 
to the format of the question or to lack of mastery of the 
concept involved, and to decide what steps to take for 
remediation. CIMS staff recommend that teachers consider options 
other than reteaching the topic, such as spiraling homework 
assignments, peer tutoring, or "Do Now" exercises in class. 

CIMS project staff also make available to school 
administrators periodic testing reports, summarizing the number 
of tests that had been administered and scored for each class in 
their school; and management reports, summarizing the percentage 
of objectives mastered on the tests by each grade. In addition, 
a computer operator's checklist, developed to keep track of the 
tests administered by each teacher, is generated weekly to help 
administrators monitor pacing of the tests. Supervisors (A.P.s 
or principals) are expected to review the test results, and to 
encourage and monitor teachers 1 use of the reports for planning 
instruction and for spiraling homework assignments. 
STAFF DEVELOPMENT 

Since implementation of the New York Citv Scope and Sequence 
requires a modification of the traditional topic-centered 
approach to teaching mathematics and a change in methods of 
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administration and supervision, staff development has been an 
intrinsic part of the CIMS system from its inception. Staff 
training provided to schools implementing PC-CIMS includes an 
introduction for school staff to the operation and use of the 
system, and formal and informal follow-up sessions aimed at 
helping teachers understand and use the test reports in their 
instructional planning, lesson design, and homework assignments. 
Staff development for principals and A.P.s deals with analyzing 
and using test report data for administrative and supervisory 
purposes, as well as other issues related to mathematics 
instruction and supervision, and is provided informally through 
ongoing consultation about the program and formally at 
districtwide staff development meetings conducted by CIMS staff 
for school administrators. In these meetings, CIMS encourages 
experienced principals and A.P.s to serve as models and guides 
for their colleagues joining the project. CIMS project staff 
also work with district mathematics coordinators to prepare them 
to assume responsibility for providing staff development and 
managing project activities. 
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III. EVALUATION METHODOLOGY 



OREA/I .S.E.U. has conducted evaluations of the CIMS-Mathematics 
project since 1982-83. These evaluations have documented the 
continued expansion of the project in New York City districts and 
schools, growth of the PC management component, ongoing revision 
of CIMS curriculum and testing materials, and overall growth in 
mathematics achievement. In 1987-88, interest in the PC 
management system and its potential for improving mathematics 
instruction and supervision led to a close examination of the 
operation uf PC-CIMS in a sample of schools in one district. The 
success of the PC model and the increasing demand for it suggest 
thafc an examination of project efforts to prepare districts and 
schools to assume responsibility for managing project activities 
is now needed. 

OPEA designed the 1988-89 evaluation, therefore, to assess 
the effectiveness of project efforts to promote the ongoing local 
management of project activities and to identify the arrangements 
and mechanisms that provide for the greatest degree of 
self-sufficiency on the part of district and school staff. The 
evaluation was in two parts. The first reviewed briefly the 
continued operation of the project in a number of districts 
implementing CIMS-Mathematics for varying lengths of time, 
focusing on efforts of district staff to adapt the program to 
meet current needs. The second part of the evaluation examined 
the operation of the project in a sample of schools in two 
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districts, one implementing POCIMS for the first time during the 



current year and one continuing operation of already existing 
systems. This part of the evaluation examined the roles of 
project, district, and school staff in can /ing out. project 
activities and preparing for the ongoing management of the 
project at the district and school levels. 
EVALUATION QUESTIONS 

The 1988-89 evaluation was therefore designed to answer a 
number of questions: 

• How is CIMS-Mathematics currently being used in districts 
that have implemented it for three to six years? What are 
the current concerns of district and school staff and how 
have project activities been adapted to meet their needs and 
resources? 

• What staff development was provided by CIMS project staff to 
support the PC-CIMS system and to train school staff in its 
use? What steps were taken to prepare district and school 
staff to take ongoing responsibility for the project, and to 
what extent did district and school staff assume this role? 
What arrangements and practices are most conducive to the 
successful operation of the project? 

• To what extent and how did principals, supervisors, and 
teachers in the PC schools make use of the tests and test 
reports? What effect did the computer management system 
have on instructional and supervisory practices in the 
schools? How can the computerized management system best be 
used, and how can these uses be effectively promoted? 

• What impact did participation in the CIMS-Mathematics 
project have on student achievement in mathematics? 

• What is the level of correlation between performance on the 
CIMS tests and performance on the citywide test of 
mathematics achievement? 
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SAMPLING 

To answer these questions, the 1988-89 evaluation of the 
CIMS-Mathematics project was divided into two parts. The first 
part of the evaluation examined the implementation of project 
activities and their impact on student achievement in a sample of 
five districts using CIMS for varying lengths of time. The 
districts were chosen to reflect the history of the project and 
to represent all models of the CIMS computer management 
component. They included three of the original districts 
involved in the project since 1982-83 (C.S.D.s 1, 18, and 32). 
These districts had implemented the original centralized computer 
systems; only C.S.D. 1 has since changed to a school-based PC 
^'stem. Also included in the evaluation were two districts which 
had joined the project more recently and implemented PC versions 
of the management system (C.S.D.s 17 and 30). This segment of 
the evaluation examined the operation of the project and its 
effect on student achievement in all grades. 

The second part of the evaluation consisted of a close 
examination of the project's activities in a sample of schools in 
the two PC districts examined in the first part of the 
evaluation. C.S.D. 30 had been chosen as one of the original 
sites for the PC project in 1986-87. As of 1988-89, the PC 
component was operational in 13 schools in the district. C.S.D. 
17 became a PC site in 1988-89 and implemented the never 
hard-disk system with expanded functions. Five schools in the 
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district were involved in the PC component, with an additional 
five schools to be added each year in future years. 

In order to examine the operation of the PC project closely 
in a number of settings, a sample of four schools was selected: 
two in C.S.D. 17 (hereafter called Schools 1 and 2) and two in 
C.S.D. 30 (Schools 3 and 4). All but School 4 were new to PC-CIMS 
in 1988-89. The schools were chosen in consultation with project 
staff to represent "best cases," places where commitment to the 
project was strong. In order to focus the investigation on a 
developmental sequence of grades that had not previously been the 
target of an evaluation, OREA examined the implementation of the 
PC project in grades one, two, and three. 
DATA SOURCES 

Evaluation of the CIMS-Mathematics project in 1988-8S was 
based on a variety of data sources: 

• In-depth phone interviews were conducted with the district 
mathematics coordinators of the districts chosen for 
examination. The interviews dealt with the continued 
implementation of project activities during the year, 
operation and use of the computer management systems and 
tests, and ongoing staff development support provided to 
project participants. Also covered were district staffs' 
perceptions of the effectiveness of project components and 
plans for future activities. Mathematics coordinators of 
the two districts selected for part two of the evaluation 
met with the OREA evaluator several times over the course of 
the year and were interviewed formally in person at the end 
of the year. 

• Extensive in-person interviews were conducted at the end of 
the school year with principals, assistant principals, and 
teachers in grades one, two, and three in the four sample 
schools in C.S.D.s 17 and 30. The interviews focused on the 
operation of the computer management systems; the nature and 
extent of staff development provided to teachers, 
principals, and supervisors; participants' use of the test 
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reports for instruction and supervision; and their 
assessment of project materials and activities. 

• Interviews were conducted with the CIMS co-directors and the 
curriculum leaders assigned to C.S. D.s 17 and 30. The 
interviews dealt with participants 1 roles and 
responsibilities, the operation of project activities, the 
degree to which project goals were achieved, and future 
plans. 

• Staff development sessions conducted by district and project 
staff were observed (principals 1 meetings in C.S. D.s 17 and 
30 district offices; formal and informal staff development 
sessions for teachers in the four sample schools) and 
project records examined (curriculum developer's logs of 
visits to PC schools; checklists of CIMS tests administered 
and scored) . 

• Mathematics achievement data for general education students 
in the five districts were analyzed. Scale scores of 
students on the New York City Mathematics Test* in spring 
1988 and 1989 were compared. Changes in student mea 
mathematics achievement levelsfrom year to year served as a 
measure of project impact. In addition, in order to 
determine the correlation between performance on the CIMS 
tests and the New York City Mathematics Test, the percentage 
of objectives mastered on the CIMS tests was compared with 
scores on the citywide test for all students in C.S.D. 30 
for whom matched scores were available. 

DESCRIPTION CF SITES AND RESPONDENTS 

The five districts chosen for the evaluation differed from 

each other in a number of ways. C.S. D.s 18 and 30 were ninth and 

tenth, respectively, among the 32 New York City school districts 

in 1988 mathematics achievement; C.S.D. 1 was 2 2nd, C.S.D x7 was 

25th, and C.S.D. 32, which had made great strides over the 

previous two years, was 19th. Of the districts examined, C.S. D.s 

1, 30, and 32 had the highest proportJ ;>n of limited English 



*This was the Metropolitan Achievement Test (M.A.T.). 1^ has 
been "customized" to correlate with the city's required scope 
and sequence of topics, which ir identical with the CIMS Scope 
and sequence. 
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proficient (LEP) students, one-fifth of their student 
populations. Percentages of students qualifying for free lunch 
(this is considered a general indicator of low socioeconomic 
level) ranged from 52 percent in C.S.D. 18 to 86 percent in 
C.S.D. 1. Inus, C.S.D. s 17 and 30, the two sites of the 1988-89 
examination of the PC-CIMS project, ranked 25th and tenth among 
the 32 community school districts in New York City in mathematics 
achievement in 1987-88, respectively. 

The C.S.D. 17 schools examined in the evaluation were very 
large and overcrowded. School 1, with a 198P register of 1,7 30 
students, was operating at 195 percent of its capacity; School 2, 
with 1,176 students, was at 135 percent capacity. In comparison, 
School 3, in C.S.D. 30, had 487 students (100 percent capacity), 
and School 4 had 734 students (77 percent capacity). All schools 
averaged 10 percent LEP students, except School 3 (17 percent) . 
The proportion of students eligible for free lunch varied from 63 
percent in School 3 to 94 percent in School 1. All four schools 
reported mathematics achievement scores above the average for 
their districts in 1988. Nevertheless, there were large 
differences in percentage of students scoring at or above grade 
level in mathematics achievement: 59 percent in School 1, 48 
percent in School 2, 70 percent in School 3, and 91 percent in 
School 4. 

Interviews chosen to include approximately half of the 
teachers on each grade level at each school were completed with 
38 of thr« 74 first, second, and third grade teachers in the four 
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sample schools. On the whole, the respondents were experienced 
teachers. They had taught an average of ten years, with only 
slight variations among the schools. Only three were in their 
first year of teaching, and two in the second. However, of the 
38 teachers, nine were teaching at their current grade 
levels for the first time. The sample included 13 first grade, 
14 second grade, and 11 third grade teachers. 
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IV. FINDINGS 



This chapter presents the findings of the 1988-89 
CIMS-Mathematics project evaluation. The findings concerning 
project activities across five districts are reported first, 
followed by the results of a more detailed examination of the 
project in four schools in two districts. A summary of student 
mathematics achievement data in the districts and schools 
concludes the chapter. 
ACROSS-DISTRICT EVALUATION 

The five districts included in the evaluation had 
participated in the project for various lengths of time and 
implemented different models of the computer management system. 
C.S.D.s 1, 18, and 32 had been involved in the project since it 
was piloted in 1983-84; C.S.D.s 17 and 30 were in their third 
and fourth year of project implementation, respectively. C.S.D.s 
18 and 32 continued to implement the original district-based 
model of the CIMS management system. C.S.D. 30 used the earlier 
version of the school-based PC-CIMS system and C.S.D. 17, the 
newer hard-disk system; C.S.D. 1 had switched from a 
district-based to a school-based system in 1987-88. In addition, 
the districts had divergent project histories and faced different 
issues in implementing project activities during the year. The 
evaluation focused only on those aspects of the project that had 
been of concern to the participants during the current year. 
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C.S.D. 1 

C.S.D. 1 has had a history of strong district commitment to 
the project, but uneven support by school administrators and some 
teacher resistance to aspects of the curriculum and testing 
system. In 1987-88, the district had switched from a centralized 
management system, which had many problems, to in-school 
microcomputers. With a new mathematics coordinator in 1988-89, 
the district mounted an effort to revive commitment to the 
project. Five schools were chosen as sites for revitalization on 
the basis of need and willingness to re-commit themselves to the 
project. In conjunction with the mathematics coordinator, CIMS 
curriculum leaders trained computer operators in these schools, 
checked manuals for the latest revisions, and led workshops for 
teachers on the curriculum and interpretation of the test 
results. At the end of the year, the district mathematics 
coordinator and CIMS staff members agreed that the effort had 
been somewhat successful, and that the schools had varied 
considerably in the extent to which they supported the effort. 
For the following year, five additional schools were to be 
selected for revitalization, with the mathematics coordinator 
continuing to oversee the original five schools. 

The mathematics coordinator *lso sought to overcome 
resistance by meeting with different groups in the district and 
attempting to change attitudes toward the project. She consulted 
with the Early Childhood unit over philosophical differences 
concerning the amount of testing in the early grades, and secured 
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their agreement to test. She also met with seventh grade 
mathematics supervisors and sought their support for the revised 
curriculum coming the following year. In general, the mathematics 
coordinator stressed the fact that CIMS is a resource in support 
of the New York City Scope and sequence and the customized 
citywide mathematics test. On the whole, she estimated that 
about half the teachers in the district used the curriculum 
manual as their main instructional resource during the current 
year. There had been some earlier resistance to the amount of 
testing the project involved, and the district had decided to use 
the unit tests without the quarterly tests. The mathematics 
coordinator monitored the progress reports showing the number of 
tests administered, and found that use was regular in some 
schools and sporadic in others. She contacted principals in 
order to encourage usage. 

For the following year, the district planned to continue to 
re-emphasize the curriculum, with staff development on the themes 
of problem-solving, thinking skills, and reasons for using CIMS. 
The mathematics coordinator planned to conduct a staff 
development session in each school as a refresher and recommended 
that workshops be held for supervisors (principals and assistant 
principals) in the district. The experience in C.S.D. 1 clearly 
shows the importance of local support of the project. 
C.S.D. 17 

In 1988-89, C.S.D. 17 was in its third year of district-wide 
implementation of the CIMS curriculum. The newer PC hard-disk 
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management system was introduced this year into five district 
schools, including one intermediate school. The project had the 
strong support of the district superintendent. A special effort 
was made in the district to coordinate funded remedial programs 
with project activities, which was facilitated by the fact that 
the mathematics coordinator was also the district coordinator of 
funded mathematics programs. Teachers from the funded remedial 
programs attended CIMS training along with the other teachers, 
and the mathematics coordinator hoped to encourage use of the 
test reports by funded teachers. 

The mathematics coordinator was responsible for staff 
development in the district apart from the five PC schools. She 
conducted staff development at the request of principals and 
supervisors, especially for special education teachers. She also 
conducted 18 hours of staff devel opment after school for grade 
six in conjunction with the O.P.C.D.'s summer training, as well 
as 14 hours of staff development for grades seven and eight. 
Both general and special education teachers attended this 
training, with especially good attendance by special education 
teachers. The focus of the training was problem exploration, 
using language skills in mathematics, and collaborative learning. 

The C.S.D. 17 mathematics coordinator reported that teachers 
in the PC schools found the pacing of the curriculum and tests 
overwhelming at first. She planned to establish a suggested 
pacing chart for the following year. At that time, the PC 
management component was to be expanded into additional schools, 
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with the mathematics coordinator taking primary responsibility 
for the first group of schools. As an additional means of 
providing for management of the project within the district, 
veteran computer operators were to be trained to act as 
troubleshooters in assisting new operators, A closer look at the 
operation of the project in selected C.S.D. 17 schools is 
provided in the following section of this report, 
C.S.D. 18 

As one of the original pilot districts, C.S.D. 18 has 
successfully implemented the CIMS-Mathematics project since 
1983-84. The mathematics coordinator reported that the 
centralized management system continued to be used effectively by 
teachers, principals, and supervisors in the district. C.S.D. 
18' s plan for 1988-89, therefore, mainly involved providing 
continued staff development in support of the project, ±n 
addition to grade six staff development in conjunction with 
O.P.C.D.'s summer training, the district conducted ongoing staff 
development for each grade level, consisting of two to three half 
days on mathematics concentration and mathematics skills. The 
mathematics coordinator also conducted four staff development 
sessions for special education teachers and professionals in the 
district, and additional staff development for selected third, 
fourth, and fifth grade teachers on using manipulatives, problem 
exploration, and thinking skills. For the following year, the 
district planned to concentrate staff development on grade seven 
to complement the . .ewly revised mathematics curriculum, and then 
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to begin again with staff development for kindergarten and the 
primary grades. 

The district also wrote a curriculum of enrichment lessons 
in thinking skills for grades five and six. The sixth grade 
curriculum began as a program for the gifted and talented and was 
then expanded to all sixth grade students. It was field tested in 
selected classes in each school during the current year. A fifth 
grade version was written during the current year, to be field 
tested during the coming year. 
C.S>D. 30 

In C.S.D. 30, PC-CIMS was in its third year of operation and 
had been expanded to 13 schools. A major effort in C.S.D. 30 was 
to encourage district self-sufficiency in managing project 
activities. At the beginning of the year, CIMS staff categorized 
project schools as A, B, or C schools based on their length of 
time in the project and readiness to be self-sufficient. Those 
identified as A schools were ready to manage project activities 
on their own; B schools were in their second year of 
implementation, but were not considered ready to be self- 
sufficient; C schools were those new to the project, and thus 
needing extensive support from the project. Eventually, CIMS 
eliminated the category of B schools, thus prompting schools to 
take over management of the project after one year. During 
1988-89, the district mathematics coordinator took responsibility 
for those schools already using the PC system, as well as all 
non-PC schools in the district, she conducted one workshop in 
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each veteran PC school to review use of the manual and test 
reports, and one workshop in each of the non-PC schools on use of 
the manual. Three days of staff development were also held twice 
during the year for sixth grade teachers. 

Coordination of the CIK J management system with funded 
programs in this district was promoted through workshops led by 
CIMS staff for teachers in funded mathematics programs. These 
teachers received copies of the management reports, but the 
mathematics coordinator reported that principals varied in the 
extent to which they promoted coordination of remedial 
mathematics programs with CIMS. 

PC-CIMS continued to be implemented successfully in C.S.D. 
30, although the mathematics coordinator reported that teachers 
in the new PC schools sometimes complained that there was too 
much material to cover and too much pressure to keep up with the 
pacing. There were also some problems with teachers in the 
district not having updated pages of the manuals, and the 
mathematics coordinator checked manuals in the schools selected 
to join PC-CIMS the following year. At that time, 18 of the 25 
district schools were expected to have the PC component. 
Additional information on C.S.D. 30 schools is reported in the 
following section on PC-CIMS. 
C.S.D. 32 

In C.S.D. 32, there has been an ongoing effort to modify the 
CIMS curriculum and tests in line with perceived district needs. 
According to the mathematics coordinator, the district made small 
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changes in the sequence of the fifth and sixth grade curricula 
after notifying teachers and supervisors of the proposed changes 
in order to achieve consensus. The district was in the process 
of u iting its own grade eight mathematics curriculum. 

The district was also completing translation of the CIMS 
tests into Spanish (which it undertook at considerable expense) , 
and sending them out for field approval. According to the 
mathematics coordinator, translation of the tests became critical 
two years ago, when for the first time there were bilingual 
versions of the citywide mathematics test. (The mathematics 
coordinator estimated that as many as 6,000 students were 
eligible for English as a second language or bilingual classes in 
the iistrict.) There had also been a longstanding concern in the 
district that the CIMS tests were too long and too infrequent for 
district students. Therefore, the district divided the tests into 
approximately twice as many shorter tests: 32 unit tests in 
grade two, 23 unit tests and four quarterly tests in grades three 
through five, and 19 unit tests and four quarterly tests in grade 
six. (CIMS project staff did not sanction or encourage modifying 
the CIMS tests in this way.) The mathematics coordinator 
monitored a test utilization chart, which showed that tests were 
used by two-thirds of teachers, on the average. According to the 
mathematics coordinator, usage was never below 33 percent, and 
for a substantial number of tests there was 90 percent usage. 

The district expanded staff development through the role of 
model mathematics lab teachers in the schools, who conducted two 
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periods of staff development for every period of instruction, and 
through the responsibilities assumed by mathematics lab, lead 
teachers in junior high schools. The district replicated the 
summer grade six staff development in six different schools and 
conducted workshops on collaborative learning. A conference for 
junior high school assistant principals on the role of language 
in mathematics instruction dealt with lesson plans and 
instructional strategies for incorporating language skills in 
mathematics. This was to be the theme for staff development the 
following year. 
EVALUATION OF PC-CIMS 

The 1988-89 evaluation sought to investigate the 
implementation of PC-CIMS in four schools in order to identify 
the arrangements that lead to its successful operation. The 
sample schools were chosen as places where commitment to the 
project was strong. The evaluation focused on the activities of 
CIMS, district, and school staff that made the system work and 
the arrangements set in motion to keep it in operation. Because 
three of the schools were new to the project, there was an 
emphasis on using the systsm, pacing the curriculum, and 
administering the tests. In addition to this, however, the 
evaluation also sought to examine the effectiveness and potential 
of the project for improving mathematics instruction and 
supervision. The activities of the CIMS project staff are 
reported first, followed by the activities of district and school 
staff. 
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Rolr of CIMS Staff 

A major objective for the year was to deepen the involvement 
of district and school staff in the operation of the program. As 
reported earlier, CIMS staff encouraged schools to become 
self-sufficient by eliminating direct support after one year on 
the project. Along with this went a redefinition of the role of 
the district mathematics coordinator, who was to assume 
responsibility for the overall operation of the project in the 
district and for staff development in schools after their first 
year of support from CIMS. 

In line with their status as first-year schools, Schools 1, 
2, and, 3 received an introduction to the project and follow-up 
from CIMS curriculum leaders. This included an introductory 
session at a centralized district site in September, followed by 
grade conferences and informal meetings in the schools after 
teachers had given tests and received test reports. The grade 
conferences showed teachers how to interpret the computerized 
reports and how to determine whether errors were due to the 
format of the question or to lack of mastery of the concept 
involved. Teachers were encouraged to use the test results for 
purposes other than reteaching, such as spiraling homework, peer 
tutoring, grouping for work with a paraprofessional , or "Do Now" 
exercises in class. While the grade conferences were devoted 
largely to interpretation of the test reports, the informal 
meetings were voluntary lunchtime sessions at which teachers 
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raised questions and discussed problems they were having with 
mathematics instruction. At these sessions, curriculum leaders 
often shared teaching strategies and ideas for incorporating 
manipulatives in the lessons. 

This pattern was modified slightly at School 3. Because of 
its small size, the CIMS curriculum leader was able to meet 
individually with teachers several times over the course of the 
year. These meetings, suggested by the principal, were attended 
by all teachers and dealt with interpretation of the test 
reports, pacing the curriculum, and preparation for the citywide 
test. Although the curriculum leader offered to provide 
curriculum assistance and to teach a sample CIMS class, these 
forms of assistance were not requested by the school. In School 
4, because of its second-year status, CIMS staff had no direct 
role in project activities, which were carried out by school and 
district staff. 

In addition to staff development for teachers, CIMS staff 
also provided support and training for principals and assistant 
principals (A.P.s) of participating schools. This included 
informal meetings with school administrators throughout the year 
on the operation of the project, and several districtwide 
meetings for principals and A.P.s. In addition, supervisors 
(principals or A.P.s) were required to attend staff development 
sessions conducted for teachers. At the district meetings, 
administrators were encouraged to share ideas, problems, and 
experiences with each other. At the meeting discussing the 
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midyear cumulative reports, CIMS staff asked administrators to 
consider the steps they were going to take as a result oi: the 
reports. School administrators were unanimous in their praise 
for the support they r reived from CIMS project staff. 

Over 80 percent of the teachers interviewed rated CIMS staff 
development as the most useful source of assistance with 
interpreting the test reports, and 80 percent of these teachers 
found this help adequate to meet their needs. A majority of 
teachers also f ^ und CIMS staff development most helpful on the 
issue of spiralii g homework, but found their school-based staff 
development more helpful with preparing for the citywide test. 
On the whole, teachen were most likely to find the CIMS staff 
development moderately helpful (55 percent) or minimally helpful 
(29 percent). Trnchers in School 3, who had received the 
greatest amount of individual ass" stance from CIMS staff, were 
the most satisfied with this help. However, over a fifth of the 
respondents in the first-year schools complained that the CIMS 
sessions were repetitious or too aneral to be of use. 
Role of District Staf f 

The two districts involved in the evaluation were led by 
superintendents who were strongly committed to the CIMS program. 
At the distr. meetings, they demonstrated familiarity with the 
project and th^ir district test reports. In general, school 
admi. istrators expressed satisfaction with district support of 
the project The only problematic issue mentioned by principals 
was the need for increased aide time for running the computer 
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systems, and this seems to have been worked out to their 
satisfaction. 

The planned role of the district mathematics coordinators, 
who were both new to their jobs, worked generally as intended. 
In the three first-year schools , the mathematics coordinators 
attended staff development sessions conducted by CIMS staff. At 
the school meetings observed, they interacted with teachers and 
offered comments and suggestions. In School 4, the mathematics 
coordinator conducted lunchtime meetings for teachers on pacing 
the curriculum and other issues, at the request of the school 
liaison. Toward the end of the year, the mathematics 
coordinators were encouraged to begin taking responsibility for 
thj project and were involved in selecting and preparing new 
schools to join the project the following year. 
P ole of School Liaisons and Supervisors 

All schools examined by the evaluation had principals 
committed to the project and exceptionally strong CIMS liaisons, 
although the situations in their schools differed somewhat. 
Schools 1 and 2 were very lcrge, and an A. P. functioned as 
primary CIMS liaison in the school. In School 1, other A.P.s 
also supervised mathematics instruction on each grade, under the 
general direction of the liaison , who was himself the supervisor 
of one grade. In School 2, the CIMS liaison supervised the CIMS 
project for the entire school , although another A. P. was general 
supervisor of the lower grades that are the focus of this report. 
Schools 3 and 4 were much smaller schools where the principal 
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supervised instruction in the lower grades. In School 3, the 
principal also acted as CIMS liaison. In School 4, a full-time 
teacher trainer (the former acting district mathematics 
coordinator) functioned as CIMS liaison. The effects of these 
different arrangements are reported in this and the following 
sections of this report. 

A major focus of this year's evaluation was on supervisors 1 
use of the computer management system. In this and the following 
sections, "supervisors" include both the CIMS liaisons and all 
other principals, and A.P.s who supervised mathematics 
instruction in the grades evaluated. 

The three first-year schools reported using faculty 
conferences, grade conferences, and individual consultations with 
teachers to follow up on the training presented by CIMS. Only in 
School 4 were formal staff development sessions conducted by 
school staff members. These were led by the CIMS liaison/teacher 
trainer, and aimed at improving teachers 1 conceptual 
understanding of mathematics. They included workshops on 
manipulatives and on testing strategies, and demonstration 
lessons on bases, metrics, and place value. 

Liaisons described their role as providing support and 
encouragement to teachers in using the curriculum. In all 
schools, liaisons reported that they reviewed the individual test 
results regularly and monitored the computer operators 1 
checklists, which indicated when specific tests were given by 
each teacher; in Schools 2 and 4 principals also reviewed the 

32 



ERLC 



43 



checklists. There was some uncertainty as to whether supervisors 
other than the liaisons reviewed the management reports. In 
School 1, the A. P. who supervised grade one reported receiving 
copies of the results, whereas the supervisor of grade two did 
not. 

Supervisory Uses of the Computer Management System . All 
supervisors said they regularly checked teachers 1 plan books for 
their pacing of the curriculum and tests, and spiraled homework 
assignments. In School 1, supervisors checked to see whether 
teachers included one skill lesson and one remediation or 
enrichment workshop per day, and if remediation activities were 
based on the management reports. Supervisors in the other 
schools reported that they could tell from observations, plan 
books, and discussions with teachers the extent to which teachers 
made use of the management reports in their instruction and 
homework assignments. general, supervisors believed that the 

main supervisory use of the management reports was co provide 
information on what was going on in Ve classrooms and to 
indicate areas requiring remediation. 

Nearly all supervisors reported that they spoke to teachers 
individually about the test results, and that teachers brought 
their reports to morthly grade conferences for discussion. The 
liaison in School 1 required teachers to bring their reports to 
their individual conferences with him after they were observed. 
He saw his role as linking teachers with others in the school vrho 
had the expertise to help them. In School 2, on the basis of the 
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midyear cumulative reports, the liaison sent letters to 
individual teachers who were behind schedule in teaching the 
curriculum, offering to meet with and assist them. This liaison, 
who supervised the project for the entire school, spoke of the 
lack of time for following up on the reports as much as he would 
have liked. 

Teachers' Reactions to Supervisors' Activities . For their 
part, teachers 1 responses provided additional documentation of 
the supervisors' activities and their effects. Nearly all 
teachers reported that their supervisors had spoken to them about 
keeping up with the pacing of the curriculum and tests. On the 
other hand, only about a third of the teachers said that their 
supervisor, had discussed specific test results with them or 
suggested instructional strategies for teaching mathematics. 
Very low findings for Schools 2 and 4 might be thought to be 
related to the fact that for teachers in some grades in these 
schools, the supervisor was a different person from the CIMS 
liaisoi who might also have followed up on the project. 
However, the same findings held across all grades examined in 
these schools. 

The findings suggested that super isors' actions were 
effective in encouraging teachers' use of the management reports. 
For all test reports, there was a tendency toward correlation 
between supervisors' practice of speaking to teachers about the 
test results and the frequency with which teachers reviewed the 
reports. For the class diagnostic report, the correlation 

34 



45 



between supervisors 1 actions and teachers 9 practice reached 
significance (p< .05): teachers whose supervisors spoke with 
them about specific test results were significantly more likely 
to review this report than those whose supervisors did not. 

Sixty-eight percent of teachers reported that their 
supervisors encouraged use of test reports for grouping students 
for instruction or homework. Despite the fact that over half the 
teachers reported that they rarely or never used the reports for 
this purpose, use of the reports appeared to be linked to 
supervisors 1 practice. Teachers who reported that their 
supervisors encouraged the use of the management reports for 
grouping students were significantly more likely (p< .05) to use 
the reports for this purpose than those whose supervisors did 
not. Use of the reports for grouping was highest in School 1, 
v/here the liaison characterized this as the primary purpose of 
the test reports. 

There were large differences among the schools regarding the 
extent to which supervisors provided concrete assistance to 
teachers in the form of suggested strategies or additional 
materials. Although nearly half of tbi teachers in School 1 
reported that their supervisors had suggested strategies for 
teaching mathematics, one-fifth or fewer teachers said this was 
the case in Schools 2 and 3. In general, teachers in Schools 1 
and 4 reported the highest levels of instructional assistance 
from their supervisors, and this appeared to be related to 
teachers' satisfaction with their school-based staff development. 
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For example, teachers in School 2, one of the large C.S.D. 
17 schools, reported the lowest levels of follow-up by 
supervisors: only ten percent reported that their supervisors 
had discussed specific test results with them, and 20 percent 
said their supervisors had suggested instructional strategies. 
Although 45 percent of all teachers interviewed reported that 
their supervisors had provided manipulative materials and 40 
percent had provided extra practice items, none of the ten 
teachers in School 2 reported receiving either of these. 
Further, teachers in School 2 were the least satisfied with their 
school-based staff development: 80 percent rated it minimally or 
not at all helpful, or said they had received no assistance. 
Since the CIMS liaison supervised the project for the whole 
school, these findings appear to support the importance of 
getting teachers 1 direct supervisors involved in project 
activities . 

On the whole, three-quarters of the teachers found their 
school-based staff development very helpful (38 percent) or 
moderately helpful (38 percent). Therp were, however, large 
differences among the schools in this regard. Teachers in School 
4 were the most satisfied with their school-based staff 
development, which is not surprising, given the presence of a 
full-time teacher trainer. Teachers in School 1, where A.P.s for 
each grade supervised mathematics instruction, also rated their 
school staff development highly. Only teachers in Schools 2 and 
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3 found the CIMS staff development more useful than their 
school -based staff development. 

In summary, then, supervisors were most likely to use the 
computer management system to monitor the pacing of the 
curriculum and tests. Supervisors reported that they observed 
whether teachers made use of the management reports in their 
instruction and homework assignments, but only one-third of 
teachers said that their supervisors had discussed specific test 
results with them. One useful idea was to ask teachers to 
include remediation activities based on their management reports 
in their lesson plans. In order to improve supervisory practices 
as intended, it was important that all supervisors became 
involved in project activities. 
Role of Teachers 

Teachers 1 Use of Testing System and Tests . Although 
three-quarters of teachers reported that they were at the 
recommended point in the curriculum at the time of the interview, 
47 reported problems in keeping up with the pacing of the 
curriculum and 63 percent had difficulty keeping up with the 
pacing of the tests. Problems with the tests were of three 
different kinds. First, five of the ten respondents in School 2 
reported problems with the testing system itself — i.e., lags in 
getting the tests cr in receiving the results. A majority of 
these teachers said at the time of the interview that the 
situation had not improved. Secondly, teachers in School 1 had 
difficulties with the school's policy of uniformly testing all 
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classes at one time because of the size of the school. This was 
a policy instituted in previous years with a computerized 
mathematics testing system the school had developed itself. Four 
teachers in School 1 reported having to wait for tests after they 
had already completed the lessons, while two others reported that 
they were being pushed too quickly to keep up with the other 
classes. These difficulties were eased somewhat when the school 
loosened its policy on administering the tests, although at the 
time of the May interviews, teachers were still citing them as 
problems. 

Thirdly, ten teachers across the schools reported that their 
students needed more time for instruction and were not ready for 
the tests at the recommended times. This included four of the 
six respondents in School 3. Teachers addressed this situation 
by adding extra lessons or double mathematics periods in some 
schools, or in some cases, by slowing down their schedules. In 
general, however, over half of the 24 teachers who had difficulty 
pacing the tests said that these problems had not been resolved. 
Unf amiliarity with the testing system and the cyclical approach 
in the first-year schools may account for some of these problems. 

CIMS records of the computer operator's checklists showed 
different patterns of test use in the schools. On the whole, 
Schools 3 and 1 made the greatest use of the CIMS tests. First 
grade teachers in all schools administered, on the average, 13 of 
the 2 n individual module tests. Second grade teachers 
administered seven of the 12 unit tests, and third grade teachers 
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gave nine of them, on average. Use of the quarterly tests varied 
from school to school, but was consistent within schools: 
teachers averaged one quarterly test in School 2, two in Schools 
1 and 4, and four in School 3. 

Teachers in Schools 2 and 3 showed somewhat erratic patterns 
of test use. First grade teachers in School 2 averaged 12 of the 
27 tests, ranging from five to 21. Often, teachers skipped tests 
and went back to give them later, sometimes for a second time. 
Several teachers stopped giving tests in March or April. In 
School 3, first grade teachers gave an average of 18 of the 27 
tests. Two of these teachers gave seven tests in May and June; 
one teacher who gave 11 tests altogr her gave seven in May, and 
several teachers went back and gave the first test at the end of 
the year. This suggests that, although teachers may have been 
using the tests for review or reinforcement, they were clearly 
not using the results for diagnostic purposes, as intended. 

Teacher's Use of Management Reports . To what extent did 
teachers review and make use of the test reports? On the whole, 
half of the teachers reported that they reviewed the item 
analysis regularly, and half reviewed the objective summary 
report. Viewed together, 60 percent of teachers regularly 
reviewed one or other, or both, of these reports. Only 16 
percent of teachers reported that they rarely or never reviewed 
either report. Two-thirds of teachers reported that they 
regularly consulted a copy of the test when interpreting the test 
results; only 6 percent said they rarely or never did this. 
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Teachers were evenly divided in reporting whether errors 
were more frequently the result of the format of the questions or 
lack of mastery of the concept involved. This seems to be a high 
percentage of errors due to the format of the questions. Fifteen 
of the 38 teachers interviewed reported that test questions were 
often wr~ded differently, or used a different format or 
vocabulaiy than the corresponding lesson. At least one teacher 
noted this with approval and said that it was important the 
students learn different ways of using a concept. However, nine 
respondents reported that test items were sometimes unclear, 
ambiguous, or confusing. The majority of these were second grade 
teachers. 

On the whole, teachers were more likely to reteach 
objectives (41 percent) as a result of the findings than to 
spiral homework assignments (17 percent) ; the remainder reported 
that they did both equally. Percentages of teachers reviewing 
the test reports were higher in Schools 3 and 4 in all cases. 

Over sixty percent of teachers in Schools 1 and 2 reported 
that they reviewed the class diagnostic report regularly (38 
percent) or sometimes (25 percent) ; 38 percent, however, said 
they rarely or never used it. The class diagnostic report is 
generated by the newer PC-CIMS program and groups students 
according to objectives missed. Use of the report was slightly 
higher at School 2 tbsn at School 1. 

How and to what extent did teachers use the management 
reports? The highest percentages of respondents reported using 
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the test results regularly for review or "Do Kow w exercises in 
class (63 percent), paraprof essionals 1 work with students (54 
percent) , and parent meetings (54 percent) . The reports were 
least used for grouping students for instruction or homework (24 
percent), individualizing instruction or homework (21 percent), 
and peer tutoring (16 percent) . Thus, although over half of the 
26 respondents who had paraprof essionals in their classrooms used 
the reports with them, these reports were not widely used for 
grouping students by the teachers themselves; 60 percent of all 
teachers reported that they rarely or never used the reports for 
this purpose. On the whole, administrators believed that 
teachers were using the reports for grouping to a much greater 
extent than teachers themselves reported, especially in Schools 2 
and 4, where 70 and 8 6 percent of teachers, respectively, said 
they rarely or never used the reports for this purpose, and where 
use of the reports with paraprofessionals was limited. 

In the two C.S.D. 17 schools where teachers received the 
class diagnostic report which grouped students by objectives 
missed, diametrically opposed findings emerged: two-thirds of 
the teachers in School 1 regularly used the test reports for work 
with paraprofessionals, while two-thirds of teachers in Schcoi 2 
never used the reports for this purpose. A similar pattern was 
found with the use of the reports for grouping: half of the 
teachers in School 1 regularly used the reports for this purpose, 
while 70 percent of teachers in School 2 rarely or never did. 
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The greater use of the reports in School 1 appeared to be related 
to the liaison's emphasis on using the system for this purpose. 

Teachers had mixed reactions to the use of the test reports 
for parent conferences. Of 19 respondents, 12 found the reports 
useful for this purpose. Teachers were most likely to appreciate 
the solid evidence of student performance that the reports 
provided. On the other hand, seven teachers reported that the 
formats of the reports were difficult for parents to understand. 
Teachers in Schools 3 and 4 reported greater use of the reports 
with parents than with teachers in School 1 and 2. In School 1, 
nearly two-thirds of teachers rarely or never used the test 
results for parent consultations. This may be related to 
teachers 1 preference for their earlier school-developed system, 
which, they reported, had a simpler parent report. 
Administrators in School 1 were planning to develop a letter for 
parents explaining the CIMS system for the following year. 

A large majority of teachers reported that they always (32 
percent) or often (47 percent) spiraled homework assignments, and 
nearly two-thirds of all teachers reported using the test reports 
for this purpose regularly (34 percent) or sometimes (29 
percent) . Of the 23 respondents in all grades who used the 
reports for spiraling homework assignments, three-quarters found 
them very useful (44 percent) or moderately useful (30 percent) . 

Across-District Diff erences in Use and Rating of Management 
Syste m. In the C.S.D. 30 schools (Schools 3 and 4), where the 
older PC-CIMS model was in place, there were consistently higher 

42 



9 

ERIC 



53 



rates of use of the test reports, apart from grouping, and higher 
ratings of the usefulness of the management system than in the 
C.S.D. 17 schools using the nev^r PC system. Teachers in the 
C.S.D. 30 schools reviewed the item analyses and objective 
summary reports more frequently (92 percent reviewed them 
regularly compared to 29 percent in C.S.D. 17), and used them 
more frequently for spiraling homework (62 percent regularly 
compared to 2 0 percent) and parent meetings (77 percent versus 40 
percent). Teachers in C.S.D. 30 were also more likely to use the 
reports with their paraprofessionals, even though C.S.D. 17 
reports grouped students by objectives needing remediation. 
C.S.D. 30 teachers also rated the usefulness of the reports for 
spiraling homework much higher (75 percent rated them very useful 
versus 9 percent in C.S.D. 17) and rated the computer management 
system in general higher than teachers using the PC model (62 
percent found it very useful against 32 percent in C.S.D. 17). 

ihese findings may be related to the size of the schools and 
the high level of assistance C.S.D. 30 teachers received from 
CIMS and school staff, although teachers in School 1 were also 
largely satisfied with the assistance they received from their 
school supervisors. This findings may also be related to the fact 
that School 4 had used the computer management system for 
a half year before the beginning of the 1988-89 school year, 
although similar results were found in both the first- and 
second-year C.S.D. 2 0 schools. The consistency of the findings 
suggests, however, that they may be due, at least in part, to 
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iifferences in the two systems and the volume of reoorts teachers 
receive with the new system. Teachers in C.S.D. 30 characterized 
the reports as "useful and specific, " "easy to consult/ 1 "giving 
good breakdown of information." Teachers in C.S.D. 17, on the 
other hand, found their reports "too complicated," providing "an 
overabundance of data," "giving x.00 much information, ■• or "more 
than I need to know." Four teachers in School 1 reported that 
they fo* nd most useful the "Scoring Summa- y Report" accompanying 
the quarterly tc t results, which lists the objectives missed by 
each student. 

Coordination of CIMS with Remedial Programs . All schools 
had made some efforts at coordinating remedial mathematics 
programs with CIMS through use of the management reports. This 
was a special concern in C.S.D. 17, where the mathematics 
coordinator was also the coordinator of funded mathematics 
programs. In this district, teachers of remedial mathematics 
programs attended CIMS staff development sessions along with the 
otner teachers. Coordination of remedial mathematics programs 
with riMS was facilitated in School 1, where a mathematics lab 
team was responsible both for operating the computer management 
system and conducting remedial mathematics instruction Tr, the 
lcwer grades, remediation was provided by full-time 
paraprof essionals, who had access to test reports, according to 
the CIMS liaison. Through this system, few teachers coordinated 
directly with the providers of remedial instruction. 
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In School 2, teachers were encouraged to speak about test 
results with mathematics lab teachers (remedial mathematics 
teachers) who were asked to work with teachers, but the principal 
believed that the remedial programs had their own curriculum, 
which was somewhat rigid. In School 3, teachers were expected to 
share the test reports with the paraprofessionals who worked with 
groups of students. About half the respondents in these schools 
reported coordinating regularly with the providers of remedial 
instruction. The one remedial mathematics teacher in School 4, 
received copies of the test reports, according to the CIMS 
liaison. 

Teachers 1 Ratings of the Computer Management System . On the 
whole, 89 percent of the teachers found the computer management 
system very useful (42 percent) or moderately useful (47 percent) 
for improving mathematics instruction. They liked the evidence 
it provided of student progress and deficiencies, and believed it 
was a good preparation for the citywide mathematics test. Of the 
total CIMS package, the management system was the most popular 
component. While teachers were most likely to rate the 
curriculum itself as good in eidressing the instructional issues 
in mathematics for their grade (46 pe^ ent), 43 percent found *t 
fair or poo; (29 and 14 percent, respectively). Teachers liked 
the strategies provided, but 12 of the 38 interviewees said that 
they were still in need of additional materials in order use the 
curriculum successfully. 
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Satisfaction with the curriculum was lowest among first 
grade teachers: 63 percent of these teachers rated the 
curriculum fair or poor. Moreover, nearly 70 percent of the 
first grade respondents cited specific problems, most notably the 
sequence of topics. Problems noted by multiple respondents, 
including first grade supervisors, included too much time spent 
on concept development at the beginning of the year; introduction 
of addition, subtraction, fractions, and numeration delayed too 
long; and the introduction of the topic money before discussing 
5s and 10s. 

In general, commitment to continue the project was high on 
the part of the teachers interviewed: 45 percent were very 
committed and 3 4 percent moderately committed. Sixty-three 
percent of 3 0 respondents said that they were more committed to 
the program than they had been the previous year; only one 
teacher was less Committed. 
STUDENT ACHIEVEMENT 

Analyses of mathematics achievement were performed on 
students 1 scores on the M.A.T. for spring 1988 and 1989. 
Achievement data are presented here in the form of both scale 
scores and normal curve equivalent (N.C.E.) scores. Scale scores 
re^jesent student performance on a continuous scale, and thus 
indicate student growth in achievement from year to year in 
relation to the student's own previous performance. N.C.E. 
scores represent student achievement in relation to a national 
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norming sample. They are, therefore, useful for comparing 
performance across districts and schools. 

Correlated t-tests were used to determine whether the 
mathematics scores of CIMS-Mathematics students were 
significantly higher in 1989 than in 1988. Because statistical 
significance is affected by sample size and does not address the 
issue of whether the achievement changes are important to the 
students' educational development, an effect size (E.S.)* is 
reported for each comparison to indicate the educational 
meaningfulness of the gain or loss, independent of the sample 
size. 

District Findings 

Comparisons of mean gains in mathematics achievement scale 
scores between spring 1988 and spring 1989 of CIMS-Mathematics 
students in the five districts examined in the evaluation are 
summarized in Table 1. This table shows an overall mean gain of 
21.0 scale score points, which represents an educationally 
meaningful gain. All districts show mean gains that are 
statistically significant, and those in C.S.D.s 1, 18, and 30 
represent large effect sizes. This means that students in these 



*The effect size is the ratio of the mean gain to the standard 
deviation of the gain. This ratno provides an index of 
improvement in standard deviation units irrespective of the 
size of the sample. An E.S. of 0.2 is considered to be a small 
E.S., 0.5 a moderate E.S., and 0.8 a large E.S. Only effect 
sizes of 0.8 and above are considered to be educationally 
meaningful, reflecting the importance of the gains to the 
students' educational development. 
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TABLE 1 



Analysis of CIMS Mathematics Scale Scores 
for All Districts: 1988-89 



Dist r 


ICt N 


Sorina 
Mean 


1988 
S . D. 


SDrincr 
Mean 


1989 
S . D. 


Difference 
Mean 0 S .D. 


E . S . 


01 


3,980 


601.3 


48.9 


624.1 


43.1 


22 .8 


27. 5 


0.8 


17 


11,62? 


601.4 


45.8 


621.0 


41.8 


19.6 


26.2 


0.7 


18 


7,381 


622.3 


51.7 


643.4 


47.9 


21.1 


27.8 


0.8 


30 


8,554 


615.3 


50.9 


640.5 


47.6 


25.2 


28. 6 


0.9 


32 


7,022 


605.8 


48.4 


622 .8 


41.0 


17.0 


27.2 


0.6 


Total 


38,559 


609.3 


49.6 


630.2 


45.4 


21.0 


27. 5 


0.8 



a Analysis of covariance showed that there were significant district 
differences on the spring 1989 scale scores using spring 1988 scores as 
a covariate [F(l,4) = 266. 0, p<.001]. Scheffe post-hoc comparison tests 
revealed that both C.S.D. 18 and C.S.D. 30 had significantly higher 1989 
adjusted mean scale scores than C.S.D.s 1, 17, and 32. 

b These mean differences were significant at p<.05. 

There was an overall statistically significant 
mean gain of 21.0 scale score points which 
represents an educationally meaningful gain. 

• Mean gains for each district represented moderate 
to large effect sizes. 
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districts made real gains in mathematics achievement that are 
educationally meaningful. 

Tables 2 through 6 present grade-by-grade analyses for each 
district (Appendix A) . As shown in these tables: 

• C.S.D. 1 showed an overall statistically significant mean 
gain of 22.8 scale score points ( S.D. -27.5) . This mean gain 
was educationally meaningful. All grades showed positive 
mean gains from 1988 to 1989. The mean gains for grades 3 
to 6 were educationally meaningful. 

• In C.S.D. 17 , there was an overall statistically significant 
mean gain of 19.6 scale score points (S.D. =26.2) , which 
represented a moderate effect size. Grades 3 through 7 
showed positive mean gains, and the gains for grades 3 
through 5 wero educationally meaningful. 

• C.S.D. 18 showed an overall statistically significant mean 
gain of 21.1 scale score points (S .D. =27. 8) . an 
educationally significant gain. All grades showed positive 
mean gains from 1988 to 1989. The mean gains for grades 3 
through 5 were educationally meaningful. 

• In C.S.D. 30, there was an overall statistically significant 
mean gain of 25.2 scale score points ( S.D. =28.6) . This was 
an educationally meaningful gain. All grades showed 
statistically significant mean gains, which in grades 3 
through 5 were educationally meaningful. 

• C.S.D. 32 showed an overall statistically significant mean 
gain of 17.0 scale score * oints ( S.D. =27.2) , representing a 
moderate effect size. All grades showed positive mean gains 
from 1988 to 1989. The mean gains for grades 3 through 5 
were educationally meaningful. 

Table 7 presents the overall district findings in terms of 
student N.C.E. scores. As seen in this table, there was an 
overall statistically significant decrease of 0.5 N.C.E. s across 
the districts, but this was not educationally meaningful. 
C.S.D.s 1 and 30 showed positive mean gains in N.C.E. scores, but 
again these were not educationally meaningful. Tables 8 to 12 
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TABLE 7 



Analysis of CIMS Mathematics N.C.E. Scores 
for All Districts: 1988-89 



District 8 N 


Spring 
Mean 


1988 
o . U. 


SDrina 
Mean 


J.989 
S . D. 


Difference 
Mean 0 S.D. 


E.S. 


01 


3,980 


52.4 


21.0 


53.2 


20.1 


0.8 


14.7 


0.1 


17 


11,622 


52.2 


21.2 


51.2 


21.1 


-1.0 


14.3 


0.1 


18 


7, 381 


63.4 


22.5 


62.8 


22.3 


-0.4 


13.3 


0.0 


30 


8, 554 


61.4 


21.7 


62.6 


22.1 


1.2 


14. 0 


0.1 


32 


7,022 


54.3 


21.1 


52.1 


20.6 


-2.2 


14.5 


0.1 


Total 


38,559 


56.8 


22.1 


56.3 


22.1 


-0.5 


14. 2 


0.0 



a Analysis of covariance showed that there were significant district 
differences on the spring 1989 N.C.E. scores using spring 1988 scores as 
a covariate [F(l,4) = 193.8, p<.001]. Scheffe post-hoc analysis 
revealed that the 1989 adjusted mean N.C.E. scores for C.S.D. 18 ard 
C.S.D. 30 were significantly higher than those obtained for C.S.D. s 1, 
17, and 32. 

^hese mean differences were significant at p<.05. 



There was an overall statistically significant 
decrease of 1.5 in N.C.E. scores throughout the 
five districts, but the decrease was not 
educationally meaningful. 

• Of the five districts, C.S.D.s 1 and 30 were 
the only two districts with positive mean N.C.E. 
gains. 
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(Appendix B) present grade-by-grade analyses for each district. 
As seen in these tables: 

• C.S.D. 1 showed an overall statistically significant gain of 
0.8 N.C.E.s. Grades 3, 5, 6 and 7 made positive mean gains 
which were statistically significant. 

• C.S.D. 17 showed an overall statistically significant mean 
decrease of 1.1 N.C.E.s, but this was not educationally 
meaningful. Grades 3, 6, and 7 showed statistically 
significant mean decreases and grades 4 and 5 made 
statistically significant mean gains. 

• In C.S.D. 18, there was an overall statistically significant 
mean decrease of 0.5 N.C.E.s, but this was not educationally 
meaningful. Grades 3, 4, and 5 showed statistically 
significant positive mean gains, and grades 6 and 7 
statistically significant me^n decreases. 

• C.S.D. 30 showed an overall statistically significant mean 
gain of 1.2 N.C.E.s. All grades but grade 6 showed 
statistically significant mean gains. The mean gain for 
grade 5 represented a moderate effect size. Only in C.S.D. 
30 did grade 7 show positive mean gains, an unusual finding. 

• In C.S.D. 32, there was an overall statistically significant 
mean decrease of 2.1 N.C.E.s, which was not educationally 
meaningful. Grades 6 and 7 showed mean decreases which were 
statistically significant. Grade 5 showed a positive mean 
gain which was statistically significant. 

On the whole, these findings show that CIMS -Mathematics 
students in the five districts made real gains in mathematics 
achievement from 1988 to 1989. However, on the whole they 
maintained their ground in relation to their peers. 
PC-CIMS Sample Schools 

Tables 13 through 16 (Appendix C) present scale score and 
N.C.E. findings in mathematics achievement for the four sample 
schools in C.S.D. s 17 and 30 that were the subject of the 
evaluation. These tables show that students in all schools and 
grades made positive statistically significant mean gains in 
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scale scores from 1988 to 1989, which were educationally 
meaningful in all cases except grade 3 in School 4. Overall 
school mean gains ranged from 26.3 scale score points ( S.D. =27.6) 
in School 4 (Table 16) to 34.1 scale score points ( S.D. =22.6) in 
School 2 (Table 14). All overall school gains were greater than 
those made by the districts as a whole. The greatest mean gains 
were made by third graders in School 2 (41.1 scale score points, 
S. D. = 22.4) and fifth graders in School < (44.2 scale score 
points, S.D.= 23.1). Effect sizes were very large in nearly all 
cases* 

Comparisons of N.C.E. scores for the four schools showed 
positive mean gains in all schools, with statistically 
significant mean gains of 3.8 N.C.E.s in School 2 and 3.6 N.C.E.s 
in School 3. In School 1, there was a small 2.6 N.C.E. mean loss 
in grade 3, which was not educationally meaningful, and an 
overall small gain of 0.7 N.C.E.s (Table 13). school 2 showed 
mean gains in all grades (Table 14). In third grade, there was a 
1.9 N.C.E. mean gain which was not statistically significant, but 
represented a very large effect size of 1.7. Overall mean N.C.E. 
gains for both Schools 1 and 2 outstripped those of C.S.D. 17 as 
a whole. 

In C.S.D. 30, School 3 showed statistically significant mean 
gains in grades 4 and 5 (Table 15). The 6.0 N.C.E. mean gain in 
grade 4 represented a moderate effect size. The 3.6 N.C.E. 
overall mean gain in School 3 was greater than the mean gain of 
1.2 N.C.E.s in C.S.D. 30 as a whole. In School 4, third graders 
showed a loss of 10.6 N.C.E.s, which represented a moderate 
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effect size (Table 16). On the other hand, fifth graders in 
School 4 made mean gains of 10.3 N.C.E.s, which was statistically 
significant and educationally meaningful. 

Correlation of CIMS Tests with Mathematics Achievement Test 

In 1987-88, O.R.E.A. conducted a pilot study of the 
correlation between student mastery of objectives on the CIMS 
tests and performance on the citywide mathematics test for 494 
second graders in C.S.D. 30. In this analysis, the percent of 
CIMS objectives mastered during 1987-88 was correlated with 
student scale scores on the spring 1988 mathematics achievement 
test. This analysis revealed a correlation of 0.73 between 
percent of CIMS objectives mastered and performance on the 
citywide mathematics test, a very high and statistically 
significant correlation. 

In 1988-89, similar analyses were performed on the scores of 
1,422 pupils in grades three through six in C.S.D. 30 for whom 
matching data were available. The percent of objectives mastered 
on the CIMS tests during 1988-89 was correlated with student 
scale scores on the spring 1989 mathematics achievement test for 
each pupil. Table 17 presents the results of this analysis. As 
seen in this table, very high and statistically significant 
correlations were found, ranging from 0.79 in grade six to 0.88 
in grade five. This meant that performance on the CIMS tests 
explained from 62 to 77 percent of the variation in student 
achievement scores. As a result, mastery of objectives on the 
CIMS tests was shown to be an extremely good predictor of success 
on the citywide test of mathematics achievement. 
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Correlation Coefficients: Assessment of the Relationship between 
Performance on the CIMS-Mathematics Test and Citywide Mathematics 
Achievement Test for Grades Trree through Six, C.S.D. 30. 



Pearson's 

Grade n r a 



3 262 0.80 

4 430 0.84 

5 262 0.88 

6 468 0.79 

8 Coef f icients were significant at the p<.05 level. 

• Correlations between performance on the CIMS-Mathematics 
tests and the citywide mathematics achievement test ranged 
from .79 in grade 6 to .88 in grade 5. These very high 
correlations were statistically significant in all cases. 
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V. CONCLUSIONS AND RECOMMENDATIONS 



CONCLUSIONS 

In 1988-89, the CIMS-Mathematics project continued its 
successful operation in the New York City public schools. The 
PC-CIMS management component was expanded into 69 schools in five 
districts, with three new districts making use of a new hard-disk 
computer system with added functions. 

A major objective of the year was to strengthen the role of 
district and school staff in the management of project 
acti/ities. In the districts and schools examined, this 
objective was largely achieved, as district math coordinators 
took on responsibility for the ongoing operation of the project 
and school administrators were involved in overseeing its local 
implementation. 

In the districts examined in the evaluation, CIMS was 
associated with continued growth in mathematics achievement from 
1988 to 1989. In the five districts, there was an overall 
statistically significant mean gain of 21.0 scale score points 
( S.D. =27.5) . an educationally meaningful gain as determined by 
effect size. Students' mean gains ranged from 25.2 scale score 
points in C.S.D. 30 ( S.D. -28.6) to 17.0 scale score points in 
C.S.D. 32 ( S.D. =27.2) . Mean gains for C.S.D.s 1, 18, and 30 were 
statistically significant and educationally meaningful. When 
compared with a national norming sample, there was a slight 
decrease of 0.5 N.C.E.s across the districts, but this was not 
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educationally significant. C.S.D.s 1 and 30 showed mean gains of 
0.8 N.C.E s ( S.D. =14.7) and 1.2 N.C.E. s ( S.D. =14.0) , 
respectively. These findings together mean that CIMS-Mathematics 
students in the five districts made real and educationally 
significant gains ^n mathematics achievement from 1988 to 1989. 
On the whole, however, they maintained their ground in relation 
to their peers in the national norm group. Greatest N.C.E. gains 
were made in two of the three districts implementing PC-CIMS. 

All sample schools examined in the evaluation showed greater 
growth in mathematics achievement than their districts as a 
whole. Overall school mean gains ranged from 26.3 ^cale score 
points in School 4 ( S.D. =27.6) to 34.1 scale score points in 
School 2 ( S. D. =22 .6) . Mean scale score gains for all schools 
were statistically significant and educationally meaningful. 
Comparison of N.C.E. scores for the four schools showed positive 
mean gains in all schools, with statistically significant mean 
gains of 3.8 N.C.E. s in School 2 ( S.D. =13 .0) and 3.6 N.C.E.s in 
School 3 (SJD^ll.5) , 

Although the four sample schools were chosen as sites where 
the project was being implemented successfully, it is not 
possible to associate the student achievement results in detail 
with the findings gathered by the evaluation. Apart from grade 
three, the student achievement data are for grades higher than 
those covered by the evaluation. greatest gains in student 

achievement were made in Schools 2 and 3, but it is difficult to 
link this to aspects of project implementation. The evaluation 

56 



ERLC 



67 



revealed the lowest levels of follow-up by school personnel in 
School 2; however, students in this school were at the lowest 
level of achievement of the four schools at the beginning of the 
project and may have benefitted most from the introduction of a 
new program. Teachers in School 3 received the most intensive 
and individualized assistance from the CIMS curriculum leader, 
which may have contributed to the success of their efforts. On 
the whole, however, the evaluation offers little basis for 
generalizing about the factors that are associated with project 
effectiveness across contexts. 

Supervisors used the management reports primarily to monitor 
the pacing of the curriculum and tests, perhaps in line with 
first-year status of the majority of the schools, and less to 
pinpoint specific test results and suggest instructional 
remediation. Teacher responses suggest that supervisors 1 actions 
had an effect: teachers whose supervisors spoke to them about 
the reports were more likely to review them, and teachers whose 
supervisors encouraged the use of the reports for grouping were 
more likely to use them for this purpose. Mixed findings on the 
use of the class diagnostic report in the two C.S.D. 17 schools 
showed that additional staff development on utilizing the 
resources that the system offers is needed. However, consistent 
differences found in the use and ratings of the older and newer 
PC systems also suggests that a rethinking of the format and 
usefulness of the newer reports may be in order. 
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Previous evaluations have called for staff development for 
supervisors; it is now possible to make this more precise. The 
present evaluation reveals the importance of getting teachers' 
direct supervisors, in addition to the CIMS liaison, involved in 
project activities, especially in large schools. There was, in 
general, a moderate response to the CIMS staff development, and 
slightly hiyher ratings were given to teachers' school-based 
staff development, which is an encouraging finding giver the aim 
of promoting management of project activities at the local level. 
However, after providing a general introduction to the management 
system for all participants, it might be a better use of time, 
and lead to more permanent change, for the CIMS curriculum 
leaders to focus staff development on training supervisors to 
work individually with teachers, rather than conducting general 
presentations for teachers on test interpretation. In this 
regard, it would have been more beneficial if the principal in 
School 3 had attended the teachers' individual sessions with the 
CIMS curriculum leader so that individualized assistance could be 
provided to teachers after direct support by CIMS was withdrawn. 

Test reports were used most regularly by teachers for review 
or "Do Now" exercises in class, paraprof essionals ' work with 
students, ind parent meetings; they were less frequently used 
for grouping students or individualizing instrucrion. Nearly 
two-thirds of the teacher" reported using the management reports 
for spiral ing homework assignments, although teachers were still 
more likely to reteach than to spiral homework on the basis of 
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the t st reports. Teachers estimated that errors on the tests 
were as likely due to the question formats as to lack of mastery 
of the concept involved. The management system miqht prove 
useful in helping teachers find ways of teaching concepts in 
different formats. There was also an unusually high percentage 
of critical comments on the sequence of the first grade 
curriculum, which should be considered when the manuals are 
updated or revised. Teachers still express a need for additional 
materials to use CIMS successfully. One useful strategy is for 
teachers to pool items on a grade. If the manuals are revised, 
CIMS may want to consider an A.P.'s suggestion to make the 
exercises in the manual different from those in the student 
workbook. 
RECOMMENDATIONS 

On the basis of the evaluation, the following 
recommendations are made: 

• Continue to expand the PC component of the project and staff 
development in support of it. 

• Focus efforts on training supervisors, as well as CIMS 
liaisons, to provide assistance with the management system 
and its use for instructional purposes. 

• Evaluate carefully the format and number of test reports and 
their usefulness for teachers and administrators; consider 
simplifying the newer PC reports. 

• Continue efforts to promote the use of the management 
reports for grouping students and individualizing 
instruction. 

• Help teachers to find alternate ways of teaching 
mathematical concepts through the use of the testing system. 

• Consider carefully teacher and supervisor comments* in 
reviewing and revising the first grade curriculum. 
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Review test formats to check for unclarity and ambiguity in 
the questions, especially in grade two. 

Continue stressing use of the tests and test reports for 
diagnostic purposes. 

Assist teachers in finding ways to share the test reports 
effectively with parents, perhaps by linking them with other 
teachers in their schools who use them successfully for this 
purpose. 



\ 



60 

71 



APPENDIX A: 

Analysis of CIMS Mathematics Scale Scores 
by District, 1988-89 
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TABLE 2 



Analysis of CIMS Mathematics Scale Scores 
for C.S.D. 1: 1988-89 



Spring 1988 Spring 1989 Difference 

Grade N Mean S.D. Mean S.D. Mean a S.D. E.S. 



3 


834 


544.7 


37.9 


585.1 


38.9 


40.4 


31.6 


1.3 


4 


775 


589.2 


38.7 


611.7 


36.1 


22.5 


27.3 


0.8 


5 


839 


607 .4 


35.4 


634 .4 


38.8 


27.0 


20.8 


1.3 


6 


759 


633 .7 


33.0 


652 .5 


16.5 


18.8 


21.8 


0.9 


7 


773 


636.0 


28.8 


639.3 


27.8 


3.3 


19.6 


0.2 


Total 


3,980 


601.3 


48.9 


624 .1 


43.1 


22.8 


27.5 


0.8 



a Thesc mean differences were significant at p<.05. 

• There was an overall statistically significant 
mean gain of 22.8 scale score points. This mean 
gain was educationally meaningful. 

• Grades 3 to 7 showed positive mean gains that 
were statistically significant. 

• Mean gains for grades 3 to 6 represented a large 
effect size. 
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TABLE 3 



Analysis of CIMS Mathematics Scale Scores 









for 


C.S.D. 17 


: 1988 


-89 






Grade 


N 


SDrina 
Mean 


1988 
S.D. 


SDrina 
Mean 


1989 
S.D. 


Difference 
Mean" S.D. 


E.S. 


3 


2,403 


556. 6 


44 . 8 


585 .9 


41.1 


29.3 


32.4 


0 9 


4 


2,234 


586.0 


36.9 


612.1 


36.9 


26.1 


23.6 


1.1 


5 


2,468 


608.7 


35.9 


636.0 


38.5 


27. 3 


22.6 


1.2 


6 


2,215 


62 8.3 


33.0 


635.6 


33.8 


7.3 


20.9 


0.3 


7 


2,302 


629.2 


29.9 


636.0 


32.0 


6.8 


18.3 


0.4 


Total 


11,622 


601.4 


45.8 


621.0 


41.8 


19.6 


26.2 


0.7 



a These mean differences were significant at p<.G5. 



There was an overall mean gain in ot 19.6 scale 

score points, which represented a moderate effect size. 

Grades 3 to 7 showed positive mean gains that were 
statistically significant. 

Mean gains for grades 3, 4, and 5 represented large 
effect sizes. Mean gains for grades 6 and 7 represented 
small effect sizes. 
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TABLE 4 



Analysis of CIMS Mathematics Scale Scores 
for C.S.D. 18: 1988-89 



Spring 1988 Spring 1989 Difference 
Grade N Mean S.D. Mean S.D. Mean 3 S.D. E.S. 



3 


1,481 


575.8 


47. S 


608.3 


48.0 


32.5 


32 4 


1.0 


4 


.1 ,484 


603.0 


45.2 


629.9 


43.7 


27. 0 


2!>. 4 


1.1 


5 


1,48? 


628. 0 


41.8 


656.3 


43.5 


28. 4 


24.8 


1.1 


6 


1,419 


65C.3 


40.3 


665.0 


40.0 


14. 8 


23.4 


0.6 


7 


1,508 


654.9 


35.4 


657.8 


38.7 


2.8 


20.0 


0.1 


Total 


7, 381 


622 .3 


51.7 


643.4 


47.9 


21.1 


27.8 


0.8 



a These mean differences were significant at p<.05. 

There was an overall statistically significant 
mean gain of 21.1 scale score points. This mean 
gain was educationally meaningful. 

Grades 3 to 7 showed positive mean gains that 
were statistically significant. 

Mean gain for grades 3, 4, and 5 represented large 
effect sizes. Mean gain for grade 6 represented 
a moderate size. 
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Analysis of CIMS Mathematics Scale Scores 









for 


C.S.D. 30 


: 1988 


-89 






blaQc 


M 
ti 


Sorinq 
Mean 


1988 
a . U. 


serine* 
Mean 


1989 
S.D. 


Difference 
Mean" S.D. 


E. S. 


3 


1,745 


563.7 


43.9 


604. 0 


45.7 


40.3 


35.0 


1.2 


4 


1,984 


601.0 


39. 3 


628.4 


40.6 


27.4 


24. 3 


1.1 


5 


1,774 


622.6 


40.7 


655.9 


44 .4 


33.4 


23.5 


1.4 


6 


1,662 


648.7 


37.2 


664.6 


40.5 


15.8 


22.6 


0.7 


7 


1, 389 


651.4 


35. 0 


655. 3 


37.5 


3.9 


20. 0 


0.2 


Total 


8,554 


615.3 


50.9 


640.5 


47.6 


25.2 


28.6 


0.9 



a These mean differences were significant at p<.05. 



There was an overall statistically significant 
mean gain of 25.2 scale score points. This mean 
gain was educationally meaningful. 

Grades 3 to 7 showed positive mean gains that were 
statistically significant. 

Mean gains for grades 3, 4, and 5 represented large 
effect sizes. Mean gain for grade 6 represented a 
modera effect size and mean gain for grade 7 
represt ted a small effect size. 
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TABLE 6 



Analysis of CIMS Mathematics Scale Scores 
for C.S.D. 32: 1988-89 







Sprincr 


1988 


Spring 


1989 


Difference 




Grade 


N 


Mean 


S . D. 


Mpan 


s n 


Mean 


S.D. 


Lj * O 


3 


1,404 


553.5 


40.8 


587.0 


38.2 


33 .5 a 


31.9 


1.1 


4 


1,336 


589.2 


37.1 


611.4 


35.7 


22.2" 


24.2 


0.9 


5 


1,444 


612.5 


37.2 


635.5 


37.2 


23 .1" 


22.2 


1.0 


6 


1,460 


632.2 


35.3 


638.1 


33.4 


5.9" 


22.4 


0.3 


7 


1, 378 


640.4 


33.1 


640.8 


31.9 


0.4 


19.2 


0.0 


Tot?l 


7,022 


605.8 


48.4 


622.8 


41.0 


17.0" 


27.2 


0.6 



a These mean differences were significant at p<.05. 

• There was an overall statistically significant 
mean gain of 17.0 scale score points, which 
represented a moderate effect size. 

Grades 3 to 6 showed positive mean gains that 
were statistically significant. 

Mean gains for grades 3, 4, and 5 represented large 
effect sizes. Mean gain for grade 6 represented a 
small effect size. 
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APPENDIX B: 



Analysis of CIMS Mathematics N.C.E. 
Scores by District, 1988-89 
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TABLE 8 



Analysis of CIMS Mathematics N.C.E. Scores 
for C.S.D. 1: 1988-89 



Spring 1988 S pring 1989 Difference 
Grade N Mean S.D. Mean S.D. Mean S.D. E.S, 



3 


834 


50.0 


23.3 


53.2 


23.4 


3.2* 


19. 3 


0.2 


4 


775 


54.9 


23.6 


55.9 


19.2 


0.9 


15.9 


0.1 


5 


839 


53.4 


19.4 


55.6 


21.7 


2 .2* 


11.9 


0.2 


6 


759 


55.7 


20.2 


57.4 


19.7 


1.7" 


12.3 


0.1 


7 


773 


47.9 


16.8 


43.8 


15.6 


-4.1 a 


11.4 


0.4 


Total 


3,980 


52.4 


21.0 


53.2 


20. 1 


0.8 8 


14.7 


0.1 



a These mean differences were significant at p<.05. 



There was an overall statistically significant 
mean gain of 0.8 in N.C.E. scores* 

Grades 3, 5, 6 and 7 showed positive mean gains 
that were statistically significant. 
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TABLE 9 



Analysis of CIMS Mathematics N.C.E. Scores 
for C.S.D. 17: 1988-89 



Spring 1988 Spring 1989 Difference 
Grade N Mean S.D. Mean S.D. Mean a S.D. E.S. 



3 


2,403 


56.4 


25.6 


53.4 


23.8 


-3.0 


19.1 


0.2 


4 


2,234 


53.6 


22.6 


56.0 


20.0 


2.4 


14.1 


0.2 


5 


2,468 


54. 1 


19. 3 


56.7 


21.1 


2.6 


12.4 


0.2 


6 


2,215 


52.2 


19. 3 


47.8 


18.7 


-4.5 


12.1 


0.4 


7 


2, 302 


44.5 


17. 1 


41.5 


17.4 


-3.0 


10.4 


0.3 


Total 


11,622 


52.2 


21.2 


51.2 


21.1 


-1.. 1 


14.3 


0.1 



a These mean differences were significant at p<.05. 

There was an overall statistically significant 
decrease of 1.1 in N.C.E. scores, but this decrease 
was not educationally meaningful. 

Grades 3, 6 and 7 showed mean decreases that were 
statistically significant. 

Grades 4 and 5 showed positive mean gains that vere 
statistically significant. 
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TABLE 10 



Analysis of CIMS Mathematics N.C.E. Scores 
for C.S.D. 18: 1988-89 



Spring 1988 Spring 1989 Difference 
Grade N Mean S.D. Mean S.D. Mean a S.D. E.S. 



3 


1,481 


67.0 


24.2 


64.8 


24.6 


2.2 


16.1 


0.1 


4 


1,484 


62.7 


25.1 


65.0 


22.0 


2.3 


13.6 


0.2 


5 


1,489 


63. 8 


21.0 


67.2 


22.0 


3.5 


11.8 


0.3 


6 


1,419 


64.7 


21.3 


63.8 


19.7 


-0.8 


12.2 


0.1 


7 


1,508 


56.7 


19.4 


53.5 


20.3 


-5.2 


10.0 


0.5 


Total 


7,381 


63.4 


22.5 


62.8 


22.3 


-0.5 


13.3 


0.0 



a These mean differences were significant at p<-05. 

There was an overall decrease of 0.5 in N.C.E. 
scores that was statistically significant, but 
the decrease was not educationally meaningful. 

Grades 3, 4, and 5 showed positive mean gains that 
were statistically significant. Grades 6 and 7 showed 
mean decreases which were statistically significant. 
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TABLE 11 



Analysis of CIMS Mathematics N.C.E. Scores 
• for C.S.D. 30: 1988-89 



Grade 


N 


Sprina 
Mean 


1988 
S . D. 


Sorina 
Mean 


1989 
S.D. 


Difference 
Mean S.D. 


E.S. 


3 


1,745 


61.7 


23.6 


63.5 


24.6 


1.8' 


19.1 


0.1 


4 


1,934 


62.4 


22 .4 


64.5 


20. 3 


2.1 8 


13.2 


0.2 


5 


1,774 


61.2 


20.5 


66.8 


22.0 


5.6" 


11.6 


0.5 


6 


1,662 


64.0 


20.8 


63.5 


20.7 


0.6 


11.7 


0.1 


7 


1,389 


56.7 


19.6 


52.5 


20. 1 


4.1 a 


10.6 


0.4 


Total 


8,554 


61.4 


2 1 .7 


62.6 


22. 1 


1.2 a 


14.0 


0.1 



a These mean differences were significant at p<.05. 



There was an overall statistically significant 
mean gain of 1.2 in N.C.E. scores. 

Grades 3, 4, 5 and 7 showed positive mean gains 
that were statistically significant. Mean gain 
for grade 5 represented a moderate effect size. 
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TABLE 12 

Analysis of CIMS Mathematics N.C.F, Scores 
for C.S.D. 32: * 38-89 



Grade 



N 



spring 1988 
Mean S.D. 



Spring 1989 
Mean S.D. 



Difference 
tfean S.D. 



E.S, 



3 


1,404 


55.1 


23.7 


54.4 


23.0 


-0.7 


19.4 


0.0 


4 


±, 336 


55.4 


22.4 


55.6 


19.4 


-0.3 


14.4 


0.0 


5 


1,444 


55.8 


19.5 


56.6 


20.8 


0.9 a 


12.0 


0.1 


6 


1,460 


54.3 


20.5 


49.5 


18.8 


-<,.8 8 


12.9 


0.4 


7 


1,378 


50.8 


19.0 


44.6 


18.0 


-6.1* 


10.7 


0.6 


Total 


7,022 


54 .3 


21.1 


52. 1 


20.6 


-2.1" 


14 .5 


0.1 



These mean differences were significant at p<.05. 

There was an overall statistically significant 
decrease of 2.1 N.C.E.s. 



Grades 6 and 7 showed mean decreases that were 
statistically significant. Grade 5 was the only 
grade that showed a positive mean gain wnich was 
statistically significant. 
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APPENDIX C: 



Analysis of CIMS Mathematics Scale Scores 
and N.C.E. Scores for Sample Schools, 1988-89 
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TABLE 13 

Analysis of CIMS Mathematics Scale Scores and N.C.E. Scores 

for School 1, C.S.D. 17: 1988-89 



Spring 1988 Spring 1989 Difference 



Grade 


N 


Mean 


S.D. 


Mean 


S.D. 


Mean 


S.D. 


E. 


Scale 


Scores : 
















3 


222 


562.4 


44.8 


594.2 


37.8 


31.8" 


28 .3 


1.1 


4 


226 


587.6 


33.7 


614.5 


35.8 


26.9" 


20.2 


1.3 


5 


236 


613. 0 


35.5 


639.2 


38.1 


26. 2" 


19 .9 


1.3 


Total 


686 


588.3 


43.5 


616.6 


41.8 


28.3" 


23.2 


1.2 


N.C.E. 


Scores: 
















3 


222 


61.3 


25.8 


58.6 


23.0 


-2.6 a 


16.1 


-0.2 


4 


226 


55.3 


21.6 


58. 0 


ro.o 


2.7 a 


12.2 


0.2 


5 


236 


57. 0 


19.1 


58.8 


21.5 


1.9' 


10.9 


0.2 


Total 


686 


57.8 


22.3 


58.5 


21.5 


0.7 


13 .5 


0.1 



a These mean differences were significant at p<.05. 

All grades in School 1 showed statistically significant 
mean gains in scale scores with very large effect sizes. 

• Grades 4 and 5 showed mean gains of 2.7 N.C.E. s and 
1.9 N.C.E.s, respectively. Ther« was an overall mean 
gain of 0.7 N.C.E.s. 
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TABLE 14 

Analysis of CIMS Mathematics Scale Scores and N.C.E. Scores 
for School 2, CSD 17: 1988-89 



Grade 


N 


Sprincr 
Mean 


1988 
S.D. 


Scrina 
Mean 


1989 
CD. 


Difference 
Mean S.D. 


E.S. 


Scale 


Scores : 
















3 


148 


528.4 


34.9 


569.5 


36.3 


41. l" 


22.4 


1.8 


4 


1j1 


575. 3 


32.7 


603 . 1 


34 .7 


27. 8 8 


7-3.1 


1.2 


5 


151 


607.6 


32.1 


640. 3 


40.2 


32. 8 a 


20.6 


1.6 


Total 


431 


5/0.5 


46.9 


604.6 


47.5 


34. 1 8 


22.6 


1.5 


• C • E • 


Scores: 
















3 


148 


42.1 


21.3 


44.0 


22.4 


1.9 


1.1 


1.7 


4 


131 


47.3 


20.0 


51.4 


18.8 


4.1 8 


13.9 


0.3 


5 


151 


55.4 


17.3 


60.8 


22.1 


5.4 8 


11.9 


0.5 


Total 


431 


48.3 


20.3 


52.1 


22.4 


3.8 s 


13.0 


0.3 



a These mean diifei ences were significant at p<.05. 

All grades in School 2 showed statistically significant 
and educationally meaningful mean gains, with an overall 
mean gain of 34.1 scale score points. 

All grades showed mean N.C.E. gains. Those in grade 3 
represented a very large effect size. There , s an 
overall mean gain of 3.8 N.C.E.s. 
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TABLE 15 



Analysis of CIMS Mathematics Scale Scores and N.C.E. Scores 
for School 3, CSD 30: 1988-89 



Grade 


N 


Sprina 
Mean 


1988 
S.D. 


Sprina 
Mean 


1989 
S.D. 


Difference 
Mean S.D. 


E.S 


Scale 


Scores: 
















3 


38 


579 . 6 


2 7.9 


624.7 


38.0 


45. 1 8 


26.4 


j 7 


4 


48 


599. 0 


35.0 


636.3 


47.3 


37. 4 8 


23.5 


1.6 


5 


59 


607.1 


21.6 


635.7 


26. i 


28.7 s 


18.9 


1.5 


6 


34 


638.8 


25.6 


656.9 


38.6 


18.1" 


20.0 


0.9 


T jtal 


179 


605. 1 


33.5 


637.5 


38.7 


32. 5 a 


23 .7 


1.4 


N.C.E. 


Scores : 
















3 


38 


73.7 


16.4 


76.7 


19.8 


3.1 


13 .7 


0.2 


4 


48 


62.1 


21.3 


68.1 


23.0 


6.0 a 


11.0 


0.5 


5 


59 


53.5 


12.2 


57.0 


15.6 


3.5* 


10.7 


0.3 


6 


34 


59. 0 


15.4 


59.8 


21.2 


0.8 


10. 7 


0.1 


Total 


179 


61. 1 


18.0 


64.7 


21.0 


3.6 a 


11.5 


0.3 



"These Mean differences were significant at p<.05. 

• All grades in School 3 showed mean gains in scale scores 
with very large effect sizes. There was an overall mean 
gain of 32.5 scale score points, which was statistically 
significant and educationally meaningful. 

• Overall, students in School 3 made mean gains of 3.6 N.C.E.s. 
Grades 4 and 5 showed statistically significant mean gains 
that represented small to moderate effect sizes. 
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TABLE 16 

Analysis of CIMS Mathematics Scale Scores and N.C.E. Scores 
for School 4, CSD 30: 1988-89 



Soring 1988 Spring 1989 Difference 



Grade 


N 


Mean 


S.D. 


Mean 


S.D. 


Mean 


S.D. 


E. 




o co ires • 
















J 




608 • o 


A A *> 

4 4 • J 


620 . 5 


A A C 

44 . o 


ii - y 


J Z • u 


u . ** 


4 


88 


619.0 


37. 1 


646.6 


38.9 


27. 6 a 


21.5 


1.3 


5 


72 


637.1 


35.2 


681. 3 


46.2 


44.2 s 


23.1 


1.9 


6 


67 


661.7 


38. 3 


686.5 


46.3 


24 .7" 


20.9 


1.2 


Total 


318 


629.1 


43.7 


655. 4 


51. 3 


26. 3 8 


27 . 6 


1.0 


N.C.E. 


Scores: 
















3 


91 


83.1 


16.5 


72.5 


21.0 


-10. 6 b 


16.9 


0.6 


4 


88 


73.2 


19.6 


73 . 6 


17.3 


0.3 


11.8 


0.0 


5 


72 


69.0 


17.9 


79. 3 


20.4 


10. 3 a 


11.9 


0.9 


6 


67 


71.4 


20.3 


73.5 


20.3 


2.1 


8.8 


0.2 


Total 


318 


74.7 


19.2 


74.5 


19.8 


0.2 


14 .9 


0.0 



•These mean differences were significant at p<.05. 



• There was an overall mean gain o:" 26.3 scale scores 
points in School 4, which was statistically significant 
and educationally meaningful. All grades showed mean 
scale scores gains that represented very large effect sizes. 

• Grade 3 showed mean losses of 10.6 N.C.E. s and grade 5 
showed mean gains of 10.3 N.C.E.s. The mean gain i.i grade 5 
was statistically significant and educationally meaningful. 
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